1. Field of the Invention
This invention relates to improvements in computer systems, networks, and the like, and more particularly to improvements in methods and apparatuses for testing computer programs and processes.
2. Relevant Background
Producers of software products generally test their products extensively to ensure proper operation of the products on several different various operating systems and computer models. As software products become more lengthy and complex, the software product tests also have become more lengthy and complex, requiring ever increasing amounts of time to perform. The development of routines to test the software products, therefore, have become of concern, especially considering the market demands of producing adequately tested software in reasonable amounts of time.
Typically, to develop a suitable test for a particular software product, a software test designer studies the product specifications. He then extracts a list of product features, and generates a number of "assertions" regarding the software product (herein the "target program" or "target product.") An assertion is, for example, a specification of the behavior of the target program, or part thereof, when it is operating properly. Thus, an assertion is a statement of how the target program is expected to operate under certain conditions, or other appropriate performance or result oriented product expectation.
For each assertion, the designer then develops a set of software "test cases." The test cases exercise as many as possible of the code paths in the target product to prove the validity of the assertions. If each assertion is proven for a particular computer model and operating system, the software product is considered to be fully tested for that particular system. Thus, for a particular software product, typically the target product is tested on various combinations of computer models and operating systems on which the product may be run.
Typically, features of the target product are not totally independent. For example, most software products are constructed using a number of subroutines, calls, branches, loops, and other program techniques. These techniques perform operations that are common to multiple independent portions of the software, and which therefore are often repeatedly used by different parts of the program in which they are incorporated. Moreover, if the target program is changed, the tester must tediously change each test case to reflect the changes.
In addition to target programs performing various particular functions, sometimes the target programs serve to install still other programs onto a computer or computer system. A test of the target program, therefore, may require an analysis of the computer after the completion of the installation process. The analysis, for instance, may ensure that the files that should have been installed were installed, or that computer configuration steps have left the computer properly configured. In addition, often the target program has various options that it performs along various pathways as a part of or in order to install the underlying program. Thus, it is often desirable for the test program to dynamically test the target program as it performs the various installation procedures of the underlying programs being installed.